Document analysis using machine learning and neural networks

ABSTRACT

A method for providing training data for a machine learning model includes: monitoring a specific user as the specific user reads electronic documents on a display to determine indications of pauses in reading for greater than a specified period of time; correlating objects on each of the displayed plurality of electronic documents to the pauses in reading; identifying features for the machine learning model based on the objects and textual analysis of each of the plurality of electronic documents; presenting information related to each identified feature to the specific user; obtaining from the specific user a descriptor defining each of the identified features and a value for each of the identified features indicating a relative importance or applicability of each of the identified features; and associating obtained descriptors and values with each of the identified features.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Despite advances in resume screening, hiring managers are deluged withresumes for which the hiring managers quickly and easily determine thatthe candidate is clearly not a match for the position. Frequently,hiring practice rules require a justification as to the reason acandidate was not selected; however, this does not improve the resumescreening process. Furthermore, a new manager may need to rely on othermore experienced managers to help screen resumes and learn how certainphrases obscure the actual experience of a candidate. These phrases maybecome obvious to one experienced with the interview process, but maynot be initially detected on the resume by one without such experience.Finally, it is difficult for any one person (e.g., a resume screener) toremember the types of candidates a particular manager would like for aparticular job when there are a number of managers or a number of jobs.

Hiring is critical to the success of an organization. Initial resumescreening is an activity that is ripe for neural network screening basedon individual and collective management supervision, where supervisionincludes the labeling of resume components as input to the neuralnetwork. Resume screening is only one example. Other types of documentanalysis may also benefit from neural network screening.

SUMMARY

Systems and methods for a machine learning neural network for scalingresume scanning and amplifying human manager analysis are provided.

According to various aspects there is provided a method for providingtraining data for a machine learning model. In some aspects, the methodmay include: displaying a plurality of electronic documents forevaluation on a display of a computing device; monitoring a specificuser as the specific user reads each of the displayed plurality ofelectronic documents on the display to determine indications of pausesin reading each of the displayed plurality of electronic documents forgreater than a specified period of time by the specific user;correlating objects on each of the displayed plurality of electronicdocuments to the pauses in reading by the specific user; identifyingfeatures for the machine learning model based on the objects and textualanalysis of each of the plurality of electronic documents; presentinginformation related to each identified feature to the specific user;obtaining from the specific user a descriptor defining each of theidentified features and a value for each of the identified featuresindicating a relative importance or applicability of each of theidentified features; associating obtained descriptors and values witheach of the identified features; obtaining from the specific user anoverall value for each of the plurality of electronic documentsindicating an overall applicability of each of the plurality ofelectronic documents with respect to specific requirements of thespecific user; associating obtained overall values with each of theplurality of electronic documents; and combining the identifiedfeatures, associated descriptors, associated values, and associatedoverall values as training data for the machine learning modelassociated with the specific user.

According to various aspects there is provided a system. In someaspects, the system may include: a plurality of machine learning modelsand a computer system. Each of the plurality of machine learning modelsmay be trained with initial training data generated by a training methodincluding, for each machine learning model: displaying a plurality ofelectronic documents for evaluation on a display of a computing device;monitoring a specific user as the specific user reads each of thedisplayed plurality of electronic documents on the display to determineindications of pauses in reading each of the displayed plurality ofelectronic documents for greater than a specified period of time by thespecific user; correlating objects on each of the displayed plurality ofelectronic documents to the pauses in reading by the specific user;identifying features for the machine learning model based on the objectsand textual analysis of each of the plurality of electronic documents;presenting information related to each identified feature to thespecific user; obtaining from the specific user a descriptor definingeach of the identified features and a value for each of the identifiedfeatures indicating a relative importance or applicability of each ofthe identified features; associating obtained descriptors and valueswith each of the identified features; obtaining from the specific useran overall value for each of the plurality of electronic documentsindicating an overall applicability of each of the plurality ofelectronic documents with respect to specific requirements of thespecific user; associating obtained overall values with each of theplurality of electronic documents; and combining the identifiedfeatures, associated descriptors, associated values, and associatedoverall values as training data for the machine learning modelassociated with the specific user.

The computer system may be configured to accept outputs from each of aplurality of neural networks executing the plurality of machine learningmodels and perform specified activities based on the outputs of each ofthe plurality of neural networks. Each machine learning model in asubset of the plurality of machine learning models may be trained usingtraining data based on requirements of a different user. At least one ofthe subset of the plurality of machine learning models may evaluate aparticular electronic document, and based on an evaluation identify atleast one aspect of the particular electronic document requiringadditional information. An output of the at least one of the pluralityof neural networks may cause the computer system to perform an activityto obtain the additional information related to the identified at leastone aspect of the particular electronic document.

According to various aspects there is provided a method for evaluatingelectronic documents using a plurality of machine learning models. Insome aspects, the method may include: training the plurality of machinelearning models for a plurality of different specific users by atraining method including, for each machine learning model: displaying aplurality of electronic documents for evaluation on a display of acomputing device; monitoring a specific user as the specific user readseach of the displayed plurality of electronic documents on the displayto determine indications of pauses in reading each of the displayedplurality of electronic documents for greater than a specified period oftime by the specific user; correlating objects on each of the displayedplurality of electronic documents to the pauses in reading by thespecific user; identifying features for the machine learning model basedon the objects and textual analysis of each of the plurality ofelectronic documents; presenting information related to each identifiedfeature to the specific user; obtaining from the specific user adescriptor defining each of the identified features and a value for eachof the identified features indicating a relative importance orapplicability of each of the identified features; associating obtaineddescriptors and values with each of the identified features; obtainingfrom the specific user an overall value for each of the plurality ofelectronic documents indicating an overall applicability of each of theplurality of electronic documents with respect to specific requirementsof the specific user; associating obtained overall values with each ofthe plurality of electronic documents; and combining the identifiedfeatures, associated descriptors, associated values, and associatedoverall values as training data for the machine learning modelassociated with the specific user; evaluating the electronic documentsby each machine learning model of the plurality of machine learningmodels based on requirements of the specific user of the plurality ofdifferent specific users used to train each of the plurality of machinelearning models; outputting a value or an indication for the electronicdocuments by each neural network executing one of the plurality ofmachine learning models; combining outputs of each neural network with acombiner; and generating a final a score or indication indicating adegree of compliance with combined requirements of the plurality ofdifferent specific users based on the combined outputs of the pluralityof neural networks.

According to various aspects there is provided a method for providingtraining data for a machine learning model. In some aspects, the methodmay include: displaying a plurality of electronic documents forevaluation on a display of a computing device; annotating electronicdocuments displayed on a display of a computing device with one or moreannotations; correlating objects on the electronic documents to at leastone of the one or more annotations; identifying features for the machinelearning model based on annotated objects and textual analysis of eachof the electronic documents; presenting information related to eachidentified feature to a specific user; obtaining from the specific usera descriptor defining each of the identified features and a value foreach of the identified features indicating a relative importance orapplicability of each of the identified features; associating obtaineddescriptors and values with each of the identified features; obtainingfrom the specific user an overall value for each of the plurality ofelectronic documents indicating an overall applicability of each of theplurality of electronic documents with respect to specific requirementsof the specific user; associating obtained overall values with each ofthe plurality of electronic documents; and combining the identifiedfeatures, associated descriptors, associated values, and associatedoverall values as training data for the machine learning modelassociated with the specific user.

Other features and advantages should be apparent from the followingdescription which illustrates by way of example aspects of the variousteachings of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects and features of the various embodiments will be more apparent bydescribing examples with reference to the accompanying drawings, inwhich:

FIG. 1 illustrates an example of visual model for a neural networkaccording to various aspects of the present disclosure;

FIG. 2 is block diagram of an example system for collecting trainingdata for a machine learning model according to various aspects of thepresent disclosure;

FIG. 3 is a flowchart illustrating a method for providing training datafor a machine learning model according to various aspects of the presentdisclosure;

FIG. 4 is a flowchart illustrating a another method for providingtraining data for a machine learning model according to various aspectsof the present disclosure;

FIG. 5 is a block diagram of an example system for evaluating electronicdocuments according to various aspects of the present disclosure; and

FIG. 6 is a flowchart of a method for evaluating electronic documentsusing a plurality of machine learning models according to variousaspects of the present disclosure.

DETAILED DESCRIPTION

While certain embodiments are described, these embodiments are presentedby way of example only, and are not intended to limit the scope ofprotection. The apparatuses, methods, and systems described herein maybe embodied in a variety of other forms. Furthermore, various omissions,substitutions, and changes in the form of the example methods andsystems described herein may be made without departing from the scope ofprotection.

In accordance with various aspects of the present disclosure, systemsand methods for a machine learning neural network for scaling resumescanning and amplifying human manager analysis are provided. Hiring iscritical to the success of an organization. Initial resume screening isan activity that is ripe for neural network screening based onindividual and collective management supervision, where supervisionincludes the labeling of resume components as input to the neuralnetwork. The machine learning neural network may recognize informationthat triggers subconscious bias, thus improving the overall diversity ofan organization. Further, using the machine learning neural network ahiring company may no longer need to rely on candidates posting resumes,but instead can reach out and search millions of resumes on databases.

Machine learning technology has wide applicability both for individualcompanies looking to hire as well as companies that exist as hiringintermediaries (e.g., LinkedIn). A hiring intermediary company could aska hiring manager subscriber to take a profiling test and based on theresulting profile could search for existing candidate resumes or bios aswell as matching new inputs to generate leads for the hiring managersubscriber.

In accordance with various aspects of the present disclosure, multiplespecial purpose neural networks, i.e., a modular neural network, may beused in combination to provide overall hiring recommendations. Theneural network may be, for example, a Long Short Term Memory (LSTM)neural network, a feedforward neural network, Radial Basis FunctionNeural Network, or another type of neural network. A machine learningmodel may exist for each individual manager.

FIG. 1 illustrates an example of visual model 100 for a neural networkaccording to various aspect of the present disclosure. Referring to FIG.1, the model 100 includes an input layer 104, a middle layer (i.e., a“hidden” layer) 106, and an output layer 108. Each layer includes somenumber of nodes 102. The nodes 102 of the input layer 104 are connectedto each node 102 of the hidden layer 106. The connections are referredto as weights 110. Each node 102 of the hidden layer 106 has aconnection or weight 110 with each node 102 of the output layer. Theinput layer 104 can receive inputs and can propagate the inputs to thehidden layer 106. A neural network implementation can include multiplehidden layers. Weighted sums computed by the hidden layer 106 (ormultiple hidden layers) are propagated to the output layer 108, whichcan present final outputs to a user.

One of ordinary skill in the art will appreciate that neural networkillustrated in FIG. 1 is merely exemplary and that different and/oradditional neural networks, for example, but not limited to, LSTM neuralnetworks, feedforward neural network, radial basis function neuralnetwork, or other types of neural networks, may be used withoutdeparting from the scope of the present disclosure.

To be useful, the neural network requires training. Training the neuralnetwork refers to the process of preparing a machine learning model tobe useful by feeding it data from which it can learn. The machinelearning model may be trained based on requirements of a particularmanager with respect to the hiring criteria of a particular position.For example, different managers may have different perspectivesregarding qualifications for the same position. Each manager may train amachine learning model with their own requirements. The manager maytrain the machine learning model by selecting a number of recentresumes, some where the candidate should definitely be interviewed andsome where the candidate should definitely not be interviewed. Thenumber of training resumes should be approximately evenly dividedbetween resumes of candidates that definitely should be interviewed andresumes of candidates that definitely should not be interviewed orbetween resumes of candidates that were hired or successful in theposition and resumes of candidates that were not hired or unsuccessfulin the position. In some embodiments, a text parsing engine maydecompose the training resumes into words, phrases, and sentences. Themachine learning models may be routinely retrained based on feedbackfrom new resumes evaluated by the machine learning models, by additionaltraining resumes evaluated by a user, by additional user input based onexperiences with hired candidates, etc.

In some embodiments, a dedicated machine learning model may evaluate therequirements or preferences of a new manager to determine a currentmanager having similar requirements or preferences. By evaluating therequirements of a new manager compared to current managers, a resumescanning machine learning model trained according to the similarrequirements may be used as a baseline machine learning model for thenew manager. When a new manager would like to use the output of a resumescanning machine learning model, the new manager may perform A/Btesting. A/B testing is a way to compare two versions of a singlevariable, e.g., two words or phrases, by testing a manager's response toword or phrase A against word or phrase B and determining which of thetwo words or phrases is more effective.

An A/B testing engine may propose various words and phrases and displaytwo words or phrases on a screen. The new manager may view the displayedwords or phrases and choose whether the word or phrase displayed on theright or on the left is more effective for a hiring situation, orwhether the words or phrases are equivalent. The results of the A/Btesting may be fed to a dedicated neural network to determine whichresume scanning machine learning model already trained for a currentmanager may be suitable as a baseline for the new manager. For example,the dedicated machine learning model may evaluate the results of the A/Btesting for the new manager and generate an output indicating a currentmanager having similar requirements to the new manager. The new managermay then begin using an instance of the resume scanning machine learningmodel used by the identified current manager. The instance of the resumescanning machine learning model used by the new manager may gradually betrained more closely to the requirements of the new manager.

In accordance with various aspects of the present disclosure, biometricdata and document content context may be used to identify features fortraining the machine learning models. For example, a pause in reading adocument may be interpreted as a user encountering a feature that shouldbe included in training data for a machine learning model. A feature mayhave an associated descriptor. For example, a feature may be “BSEE” or“MSME” and the associated descriptor for the feature may be “collegedegree.” Different methods, for example, but not limited to, eyemovement tracking, touch-sensitive input, audio annotation, etc., may beused individually or in combination to identify and provide trainingdata for a machine learning model. Training data may be generated by aplurality of different users for a plurality of different machinelearning models based on requirements of the different users. FIG. 2 isa block diagram of an example system 200 for collecting training datafor a machine learning model in accordance with various aspects of thepresent disclosure. Referring to FIG. 2, the system 200 may include acomputer 210, a storage device 220, a display device 230, a user inputdevice 240, and an eye movement tracking device 250.

The computer 210 may be, for example, a laptop computer, a desktopcomputer, or other mini- or micro-computer. In some embodiments, thecomputer 210 may be or may include a cloud computing platform. Theplurality of machine learning models may be executed on the computer 210or may be executed on a different computer. The storage device 220 maybe a hard disk drive, optical disk drive, or other non-transitorycomputer-readable storage medium. The plurality of machine learningmodels may be stored on the storage device 220. The display device 230may be a video display device capable of displaying electronic documentsrendered by the computer 210. The display device 230 may include atouch-sensitive display screen configured to determine a position of afinger or stylus making contact with the touch-sensitive display screen.The display device 230 may further be configured to provide hapticfeedback based on an amount of pressure exerted on the touch-sensitivedisplay screen by a finger or stylus.

The user input devices may be one or more of a keyboard, a mouse, atrackball, a joystick, a touch-sensitive display screen, an eye movementtracking device, a stylus, a microphone, a camera, or combinationsthereof. The eye movement tracking device 250 may be a non-contact,optical eye movement tracking device or other type of eye movementtracking device.

In some embodiments, eye movement tracking may be used to identifyobjects (e.g., words, phrases, pictures, etc.) on a document that may beincluded as training data for a particular user. Eye movement trackingis the process of measuring either the point of gaze (where one islooking) or the motion of an eye relative to the head. An eye tracker isa device for measuring eye positions and eye movement. Some eyetrackers, for example video-based eye trackers or other eye movementtracking technologies, use non-contact, optical methods (e.g., reflectedlight) for measuring eye motion. Based on the light reflected off auser's eyes, a point of regard on a surface, for example the surface ofa computer display, can be computed by software.

In accordance with various aspects of the present disclosure, adocument, for example, a job application, a resume, an article, etc.,may be rendered in electronic form on a display (e.g., the displaydevice 230 of a computer system (e.g., the computer system 200) by acomputer (e.g., the computer 210). An eye movement tracking device(e.g., the eye movement tracking device 250) may track the eye movementsof a user as the user reads the displayed electronic document and sendeye movement tracking information to the computer. Based on the eyemovement tracking information, software executing on the computer maycompute the position of the user's gaze on the display device.

The system may detect pauses in the user's reading of the electronicdocument. The computer may determine based on the eye movement trackinginformation that the user's eyes have not moved for a period of time.For example, the computer may repeatedly receive approximately the sameeye movement tracking information indicating that the user's gaze hasremained in substantially the same position on the display device. Basedon the tracking information received from the eye movement trackingdevice, the computer may determine that the user has paused reading at aparticular location on the display device for a specified period oftime, for example, five seconds or another period of time. In accordancewith various aspects of the present disclosure, detection of repeatedeye movement may be used identify features of a document that may beincluded as training data for a particular user. For example, detectingeye movements repeatedly between a “skills” section and an “experience”section of a resume to compare skills with experience may identifyfeatures for the training data.

The computer may correlate the location on the display device to aportion of the displayed electronic document with a word, phrase,picture, or other object displayed on the electronic document. Thecomputer may perform a textual analysis on the electronic document andidentify features for the machine learning model based on the detectedobjects and the textual analysis. For example, when the electronicdocument is a resume or job application the computer may identifyfeatures related to logical skill development information, careerprogression information, misleading phrases related to employmentexperience, gaps in employment history, and/or other relevant features.Associated values for the features may include specific skills, specificposition levels (e.g., programmer, lead analyst, supervisor, etc.),specific phrases, number and/or length of employment gaps, etc.

When features are identified, the computer may present information, forexample via the display device, regarding the identified features andrequest additional user input regarding the identified features. Theadditional user input may be a descriptor that defines the identifiedfeature and a value or an indication of the importance or applicabilityof the identified feature. For example, a feature may be “BSEE” or“MSME” and the associated descriptor for the feature may be “collegedegree.” The value or indication may be, for example, a numeric value ona scale of one to ten or another indication of the relative importanceor applicability of the identified feature. The computer may associatethe value with the identified feature. The identified features,descriptors, and the additional user input (i.e., the values orindications) may be combined as training data for the machine learningmodel. A plurality of features may be identified in each electronicdocument; however, not all of the electronic documents may contain thesame features. Some of the same (i.e., common) features may beidentified in each of the electronic documents and some differentfeatures may be identified among the electronic documents. Both thecommon features and the different features may be included in thetraining data.

Additional user input may also be applied to an overall electronicdocument to provide a value with respect to specific requirements of theuser (e.g., a specific manager). In the context of a job application orresume, the resume may meet some, but not all, of the requirements amanager placed on a particular position. In accordance with variousaspects of the present disclosure, a value may be associated with theresume to provide an indication of how close the resume is to meetingthe requirements. For example, a scale of one to ten, or anotherrelative indication may be used to assign the value. Thus, for a resumeof a candidate that comes close to meeting the job requirements, themanager may assign an overall value of eight to the resume. Conversely,for a resume of a candidate that meets only a few of the jobrequirements, the manager may assign an overall value of 3 to theresume. The computer may associate the value assigned by the managerwith the overall electronic document and the value assigned to theoverall electronic document may also become part of the training datafor the machine learning model.

In accordance with various aspects of the present disclosure, a valueassociated with an identified feature may be an indication ofunintentional bias. The indication of unintentional bias may beidentified and mitigated in subsequent document evaluations performed bythe machine learning model. For example, to address unintentional biasvarious experts such as human resources personnel, members of protectedclasses, advocates for underrepresented groups, and legal professionalsversed in discrimination law may evaluate resumes specifically fortrigger phrases, words, grammatical constructions, etc., or specificfacts such as attendance at a college or university historicallyperceived as being targeted to a specific race or gender, to identifyitems or patterns that might be triggers for bias. As an example, acandidate for whom English is not their primary language may havetrouble with plurals and gender pronouns in the English language.However, there are many positions for which these minor grammaticalerrors may not be critical, such as a position in an engineering lab,although there are some positions where they are relevant, such aspublic relations. An expert might identify an occurrence of a linguisticcharacteristic as a potential trigger for unconscious bias. When amanager evaluating a resume shows attention to a minor detail of amissing ‘s’ in a word and rejects the resume or scores it lower, thesystem can find that correlation with the identified items or patternsindicating bias and suggest a score of potential bias. Since bias isfrequently subtle and often unconscious many examples of different typesof bias indications may be necessary to determine a manager is lookingat characteristics that may not be relevant to a job. Thus, the systemmay provide the ability to unobtrusively mitigate effects of subtlebiases on evaluations.

In some embodiments, the display device may be a touch-sensitive displaydevice and a finger or stylus position on the touch sensitive displaydevice may be used to identify objects on a document that may beincluded as training data for a particular user. In an embodiment havinga touch-sensitive display device, the user may follow the text of eachdisplayed electronic document across the touch-sensitive display with afinger or a stylus in contact with the touch-sensitive display as theuser reads each displayed electronic document.

The system may detect pauses in the user's reading of the electronicdocument based on the movement of the finger or the stylus. The computermay determine based on the touch information received from thetouch-sensitive display device that the user's finger or the stylus havenot moved for a specified period of time. For example, the computer mayrepeatedly receive approximately the same touch position informationindicating that the user's finger or the stylus has remained in contactat substantially the same position on the touch-sensitive display deviceand may determine a length of time the contact persists. Based on thetouch position information received from the touch-sensitive displaydevice, the computer may determine that the user has paused reading thedisplayed electronic document at a particular location on thetouch-sensitive display device for a specified period of time, forexample, five seconds or another period of time. In accordance withvarious aspects of the present disclosure, detection of repeated fingeror stylus movement may be used identify features of a document that maybe included as training data for a particular user. For example,detecting finger or stylus movements repeatedly between a “skills”section and an “experience” section of a resume to compare skills withexperience may identify features for the training data.

The computer may correlate the location on the touch-sensitive displaydevice to a portion of the displayed electronic document with a word,phrase, picture, or other object displayed on the electronic document.The computer may perform a textual analysis on the electronic documentand identify features for the machine learning model based on thedetected objects and the textual analysis. For example, when theelectronic document is a resume or job application the computer mayidentify features related to logical skill development information,career progression information, misleading phrases related to employmentexperience, gaps in employment history, and/or other relevant features.

In some embodiments, the display device may be a touch-sensitive displaydevice configured to provide haptic feedback in response to a specifiedamount of pressure exerted on the touch-sensitive display device by afinger or a stylus. In an embodiment having a touch-sensitive displaydevice that provides haptic feedback, the user may follow the text ofeach displayed electronic document across the touch-sensitive displaywith a finger or a stylus in contact with the touch-sensitive display asthe user reads each displayed electronic document.

The system may detect pauses in the user's reading of the electronicdocument based on the pressure exerted by the finger or the stylus tocause the touch-sensitive display to generate haptic feedback for aspecified period of time. The computer may determine based on the touchinformation received from the touch-sensitive display the location onthe touch-sensitive display of the user's finger or the stylus causingthe haptic feedback. For example, the computer may receive touchposition information indicating that the user's finger or the stylus hasexerted pressure in substantially the same position on thetouch-sensitive display device to cause haptic feedback and maydetermine a length of time the haptic feedback persists. Based on thetouch position information received from the touch-sensitive displaydevice and the length of time the haptic feedback persists, the computermay determine that the user has paused reading the electronic documentat a particular location on the touch-sensitive display device for aspecified period of time, for example, five seconds or another period oftime. In accordance with various aspects of the present disclosure,detection of repeated finger or stylus movement to cause haptic feedbackmay be used identify features of a document that may be included astraining data for a particular user. For example, detecting finger orstylus movements to cause haptic feedback repeatedly between a “skills”section and an “experience” section of a resume to compare skills withexperience may identify features for the training data.

The computer may correlate the location on the touch-sensitive displaydevice to a portion of the displayed electronic document with a word,phrase, picture, or other object displayed on the electronic document.The computer may perform a textual analysis on the electronic documentand identify features for the machine learning model based on thedetected objects and the textual analysis. For example, when theelectronic document is a resume or job application the computer mayidentify features related to logical skill development information,career progression information, misleading phrases related to employmentexperience, gaps in employment history and/or other relevant features.

In accordance with various aspects of the present disclosure, displayedelectronic documents may be annotated by a user to identify features fortraining data. A user may mark up an electronic document displayed on adisplay device (e.g., the display device 230) capable of accepting userinput using, for example, a finger, a stylus, or other suitable userinput device, to identify features on the electronic document.Standardized mark-up symbols may be defined, for example, underliningobjects (e.g., words, phrases, pictures, etc.) on the electronicdocument to indicate positive aspects, strike-through to indicatenegative aspects, circling aspects of the electronic document requiringadditional information etc. One of ordinary skill in the art willappreciate that the described mark-up symbols are merely exemplary andthat more, fewer, and/or different mark-up symbols may be definedwithout departing from the scope of the present disclosure. Inaccordance with various aspects of the present disclosure, if a standardmark-up symbol is not used, the computer may query the user, for exampleby displaying a message on the display, to determine the meaning of themark-up.

In some embodiments, the annotations may be audio annotations using aspeech recognition engine. The audio annotations may be associated withobjects on the electronic document. Standardized phrases may be used toindicate positive and negative aspects of the electronic document oridentifying aspects of the electronic document requiring additionalinformation.

The computer (e.g., the computer 210) may correlate the objects on theelectronic document to the annotations and interpret the meaning of theannotations with respect to the objects. The computer may perform atextual analysis of the electronic documents. Based on the textualanalysis and annotated objects the computer may identify features to beincluded in the training data for the machine learning model.

When features are identified, the computer may present information, forexample via the display device, regarding the identified features andrequest additional user input regarding the identified features. Theadditional user input may be a descriptor that defines the identifiedfeature and a value or an indication of the importance or applicabilityof the identified feature. For example, a feature may be “BSEE” or“MSME” and the associated descriptor for the feature may be “collegedegree.” The value or indication may be, for example, a numeric value ona scale of one to ten or another indication of the relative importanceor applicability of an identified feature to a specific employmentposition as interpreted by a particular user. A similar user input valueor indication may also be applied to an overall electronic document. Theidentified features and the additional user input may be combined astraining data for the machine learning model. In some cases, a valueassociated with an identified feature may be an indication ofunintentional bias. The indication of unintentional bias may beidentified and mitigated in subsequent document evaluations performed bythe machine learning model. For example, items or patterns previouslyidentified by experts as being triggers for bias may be correlated witha value associated with an identified feature and a score of potentialbias may be suggested, thereby mitigating the effects of subtle biaseson the evaluation.

FIG. 3 is a flowchart illustrating a method 300 for providing trainingdata for a machine learning model in accordance with various aspects ofthe present disclosure. Referring to FIG. 3, at block 310 an electronicdocument may be displayed on a display device (e.g., the display device230 of a computer system (e.g., the computer system 200). The electronicdocument may be, for example, but not limited to, an employmentapplication, a resume, a published article, etc.

At block 320, when a user is reading a document a pause in the user'sreading may be detected. The pause in reading the document may beinterpreted as a user encountering a feature that should be included intraining data for a machine learning model. In some embodiments, thepause in reading may be detected by an eye movement tracking device(e.g., the eye movement tracking device 250). The computer may receiveeye movement tracking information from the eye movement tracking device.Based on the tracking information received from the eye movementtracking device, the computer may determine that the user has pausedreading at a particular location on the display device for a specifiedperiod of time, for example, five seconds or another period of time. Inaccordance with various aspects of the present disclosure, detection ofrepeated eye movement may indicate that the user has paused reading. Forexample, detecting eye movements repeatedly between a “skills” sectionand an “experience” section of a resume to compare skills withexperience may be interpreted by the computer as a user encountering afeature that should be included in training data.

In some embodiments, the pause in reading may be detected by a finger orstylus position on the touch sensitive display device. A user may followthe text of each displayed electronic document across thetouch-sensitive display with a finger or a stylus in contact with thetouch-sensitive display as the user reads each displayed electronicdocument. The computer may repeatedly receive approximately the sametouch position information from the touch-sensitive display device anddetermine based on the touch information that the user's finger or thestylus have not moved for a specified period of time. Based on the touchposition information received from the touch-sensitive display device,the computer may determine that the user has paused reading thedisplayed electronic document at a particular location on thetouch-sensitive display device for a specified period of time, forexample, five seconds or another period of time. In accordance withvarious aspects of the present disclosure, detection of repeated fingeror stylus movement may indicate that the user has paused reading. Forexample, detecting finger or stylus movements repeatedly between a“skills” section and an “experience” section of a resume to compareskills with experience may be interpreted by the computer as a userencountering a feature that should be included in training data.

In some embodiments, the pause in reading may be detected by hapticfeedback provided by a touch-sensitive display device. A user may followthe text of each displayed electronic document across thetouch-sensitive display with a finger or a stylus in contact with thetouch-sensitive display as the user reads each displayed electronicdocument. The touch-sensitive display may generate haptic feedback for aspecified period of time as a result of the pressure exerted by thefinger or the stylus on the touch-sensitive display. The computer mayreceive touch position information indicating that the user's finger orthe stylus has exerted pressure in substantially the same position onthe touch-sensitive display device to cause haptic feedback and maydetermine that the user has paused reading the electronic document at aparticular location on the touch-sensitive display device for aspecified period of time, for example, five seconds or another period oftime. In accordance with various aspects of the present disclosure,detection of repeated finger or stylus movement to cause haptic feedbackmay indicate that the user has paused reading. For example, detectingfinger or stylus movements to cause haptic feedback repeatedly between a“skills” section and an “experience” section of a resume to compareskills with experience may be interpreted by the computer as a userencountering a feature that should be included in training data.

At block 330, the computer may correlate the location on the displaydevice to a portion of the displayed electronic document with a word,phrase, picture, or other object displayed on the electronic document.At block 340, the computer may perform a textual analysis on theelectronic document and identify features for the machine learning modelbased on the detected objects and the textual analysis. In accordancewith various aspects of the present disclosure, different features maybe identified for different machine learning models created by differentusers (e.g., managers). When the electronic document is a resume or jobapplication the computer may identify features related to logical skilldevelopment information, career progression information, misleadingphrases related to employment experience, gaps in employment history,and/or other relevant features.

At block 350, when features of the electronic document are identified,the computer may present information to the user, for example via thedisplay device, regarding the identified features and may requestadditional input from the user regarding the identified features. Atblock 360, additional user input may be obtained. The additional userinput may be a descriptor that defines the identified feature and avalue reflecting the importance or applicability of the identifiedfeature to a specific employment position as interpreted by a particularuser. For example, a feature may be “BSEE” or “MSME” and the associateddescriptor for the feature may be “college degree.” The value orindication may be, for example, a numeric value on a scale of one to tenor another indication of the relative importance or applicability of theidentified feature. The computer may associate the value with theidentified feature. A similar user input value or indication may also beapplied to an overall electronic document. The computer may associatethe value assigned by the manager with the overall electronic document.In some cases, a value associated with an identified feature or anoverall value associated with an electronic document may be anindication of unintentional bias. The indication of unintentional biasmay be identified and mitigated in subsequent document evaluationsperformed by the machine learning model. For example, items or patternspreviously identified by experts as being triggers for bias may becorrelated with a value associated with an identified feature or anoverall value and a score of potential bias may be suggested, therebymitigating the effects of subtle biases on the evaluation.

At block 370, the identified features and the additional user input maybe combined as training data for the machine learning model. Forexample, the identified features, descriptors, and associated values aswell as the value assigned to the overall electronic document may becombined as training data for the machine learning model. A plurality ofdifferent users may train a plurality of different machine learningmodels using the method.

It should be appreciated that the specific steps illustrated in FIG. 3provide a particular method for providing training data for a machinelearning model according to an embodiment. Other sequences of steps mayalso be performed according to alternative embodiments. For example,alternative embodiments may perform the steps outlined above in adifferent order. Moreover, the individual steps illustrated in FIG. 3may include multiple sub-steps that may be performed in varioussequences as appropriate to the individual step. Furthermore, additionalsteps may be added or removed depending on the particular applications.One of ordinary skill in the art would recognize many variations,modifications, and alternatives.

The method 300 may be embodied on a non-transitory computer readablemedium, for example, but not limited to, a memory or othernon-transitory computer readable medium known to those of skill in theart, having stored therein a program including computer executableinstructions for making a processor, computer, or other programmabledevice execute the operations of the methods.

FIG. 4 is a flowchart of a another method 400 for evaluating electronicdocuments using a plurality of machine learning models in accordancewith various aspects of the present disclosure. Referring to FIG. 4, atblock 410 an electronic document may be displayed on a display device(e.g., the display device of a computer system (e.g., the computersystem 200). The electronic document may be, for example, but notlimited to, an employment application, a resume, a published article,etc.

At block 420 the displayed electronic document may be annotated. In someembodiments, a user may mark up an electronic document displayed on adisplay device (e.g., the display device 230) capable of accepting userinput using, for example, a finger, a stylus, or other suitable userinput device, using defined symbols to identify features on theelectronic document. In some embodiments, the annotations may be audioannotations using a speech recognition engine. The audio annotations maybe associated with objects on the electronic document. Standardizedphrases may be used to indicate positive and negative aspects of theelectronic document or identifying aspects of the electronic documentrequiring additional information.

At block 430 objects on the displayed electronic document may becorrelated to the annotations. The computer (e.g., the computer 210) maycorrelate the objects on the electronic document to the annotations andinterpret the meaning of the annotations with respect to the objects. Atblock 440 features of the electronic document may be identified. Thecomputer may perform a textual analysis on the electronic document andidentify features for the machine learning model based on the detectedobjects and the textual analysis. When the electronic document is aresume or job application the computer may identify features related tological skill development information, career progression information,misleading phrases related to employment experience, gaps in employmenthistory, and/or other relevant features.

At block 450 information related to the identified features may bepresented to the user. The computer may present information to the user,for example via the display device, regarding the identified featuresand may request additional input from the user regarding the identifiedfeatures. At block 460 additional user input may be obtained. Theadditional user input may be a descriptor that defines the identifiedfeature and a value reflecting the importance or applicability of anidentified feature to a specific employment position as interpreted by aparticular user. For example, a feature may be “BSEE” or “MSME” and theassociated descriptor for the feature may be “college degree.” The valueor indication may be, for example, a numeric value on a scale of one toten or another indication of the relative importance or applicability ofthe identified feature. The computer may associate the value with theidentified feature.

A similar user input value or indication may also be applied to anoverall electronic document. The computer may associate the valueassigned by the manager with the overall electronic document. In somecases, a value associated with an identified feature or an overall valueassociated with an electronic document may be an indication ofunintentional bias. The indication of unintentional bias may beidentified and mitigated in subsequent document evaluations performed bythe machine learning model. For example, items or patterns previouslyidentified by experts as being triggers for bias may be correlated witha value associated with an identified feature or an overall value and ascore of potential bias may be suggested, thereby mitigating the effectsof subtle biases on the evaluation.

At block 470 the identified features and additional user input may becombined as training data for the machine learning model. For example,the identified features, descriptors, and associated values as well asthe value assigned to the overall electronic document may be combined astraining data for the machine learning model. A plurality of differentusers may train a plurality of different machine learning models usingthe method.

It should be appreciated that the specific steps illustrated in FIG. 4provide a particular method for providing training data for a machinelearning model according to an embodiment. Other sequences of steps mayalso be performed according to alternative embodiments. For example,alternative embodiments may perform the steps outlined above in adifferent order. Moreover, the individual steps illustrated in FIG. 4may include multiple sub-steps that may be performed in varioussequences as appropriate to the individual step. Furthermore, additionalsteps may be added or removed depending on the particular applications.One of ordinary skill in the art would recognize many variations,modifications, and alternatives.

The method 400 may be embodied on a non-transitory computer readablemedium, for example, but not limited to, a memory or othernon-transitory computer readable medium known to those of skill in theart, having stored therein a program including computer executableinstructions for making a processor, computer, or other programmabledevice execute the operations of the methods.

FIG. 5 is a block diagram of an example system 500 for evaluatingelectronic documents in accordance with various aspects of the presentdisclosure. Referring to FIG. 5, system 500 may include a plurality ofneural networks 510 a-510 n and a computer system 520. The computersystem 520 may include a computer, a storage device, a display device,and one or more user input devices (not shown). The computer may be, forexample, a laptop computer, a desktop computer, or other mini- ormicro-computer. In some embodiments, the computer may be or may includea cloud computing platform.

In some embodiments one or more of the plurality of neural networks 510a-510 n may be part of the computer system 520. In some embodiments thecomputer system 520 may be a computer system separate from the pluralityof neural networks 510 a-510 n. Each of the plurality of neural networks510 a-510 n may execute a machine learning model 515 a-515 n. Each ofthe machine learning models 515 a-515 n may be trained with trainingdata based on requirements of a different user.

At least one of the machine learning models 515 a-515 n may be trainedusing training data based on requirements applicable to the subset ofthe machine learning models 515 a-515 n. For example, when the machinelearning models are trained to evaluate resumes for specific employmentopenings according to requirements of different users, one of themachine learning models may be trained to evaluate resumes based ongeneral hiring requirements of the company. Resumes may be evaluatedsequentially, first based on the general requirements, then by thespecific requirements.

While the above example has been described using one machine learningmodel trained to evaluate resumes based on general hiring requirementsbefore being evaluated for specific requirements, embodiments are notlimited to this implementation. In accordance with various aspects ofthe present disclosure, more than one machine learning mode may betrained based on common requirements, for example, but not limited to,minimum education level, minimum skill level for a category ofpositions, etc., to sequentially evaluate electronic documents. Forexample, continuing with the resume evaluation example, in addition to amachine learning model trained based on general hiring requirements,another machine learning model may be trained based on hiringrequirements for a specific division of the company, another machinelearning model may be trained based on hiring requirements for aparticular department within the division, etc. Resumes may be evaluatedsequentially by each of the machine learning model. One of ordinaryskill in the art would recognize many variations, modifications, andalternatives.

In accordance with various aspects of the present disclosure, thecomputer system 520 may perform an activity based on an output 517 a-517n of at least one of the neural networks 510 a-510 n. Each of the neuralnetworks 510 a-510 n may execute a machine learning model 515 a-515 nbased on the requirements of different users to evaluate electronicdocuments. Based on the evaluations, one or more of the neural networks510 a-510 n may output aspects of the electronic documents requiringadditional information to the computer system 520. The computer system520 may perform an activity to obtain the additional information. Forexample, a machine learning model may evaluate a resume and identifygaps in employment history or other missing information. In some cases,a candidate may be asked to supply information regarding something notfound on the resume but is found on other similar resumes. The computersystem 520 may perform an activity to obtain the additional informationfrom the candidate.

Upon receiving the output from the neural network, the computer systemmay generate an information request (e.g., an email) to the candidatethat submitted the resume to obtain the required additional information.One of ordinary skill in the art will appreciate that these aspects aremerely exemplary and that other aspects may be identified withoutdeparting from the scope of the present disclosure. For example, theelectronic document may be an article, and the author of the article maybe contacted.

When the requested additional information is received by the computersystem 520 (e.g., by a return email from the candidate), the computersystem 520 may output 522 the additional information as additionaltraining data for the at least one of the plurality of machine learningmodels that identified the aspect requiring the further information. Theelectronic document may be reevaluated using the machine learning modelmodified with the additional training data provided by the additionalinformation. The machine learning model may generate a value or anindication for the electronic document and the neural network may output524 the value or indication. The value or indication may indicate adegree of compliance with the requirements of the specific user.

Each of the plurality of machine learning models may be configured todetermine that the specified requirements used to evaluate an electronicdocument applied to one or more identified features of the electronicdocument indicates an unintentional bias. The indications ofunintentional bias may be identified and mitigation during subsequentelectronic document evaluation. For example, items or patternspreviously identified by experts as being triggers for bias may becorrelated with a value associated with an identified feature and ascore of potential bias may be suggested, thereby mitigating the effectsof subtle biases on the evaluation.

In accordance with various aspects of the present disclosure, documentscreening and interviewing may be combined. An interactive electronicinterview process, for example, via Skype, WebEx, or similar interactiveaudio-video service, involving a manager and candidate at the same timemay take place with an intermediary avatar displayed to remove certaincharacteristics such as race, gender, national origin, or perceivedattractiveness from the manager's consideration. In such a scenario, themanager may review the candidate's electronic resume on the display andthe system may note that the manager pauses and reviews certaininformation, for example, the employment dates of the candidate. With orwithout a verbal annotation, the system may ask the candidate to explainthe related aspects of the resume. In addition to or alternative to averbal response from the candidate, the system may provide aspeech-to-text conversion of the response and display the supplementalinformation provided by the candidate on the document.

FIG. 6 is a flowchart of a method 600 for evaluating electronicdocuments in accordance with various aspects of the present disclosure.Referring to FIG. 600, at block 610, a plurality of machine learningmodels may be trained. The plurality of machine learning models (e.g.,the machine learning models 515 a-515 n) may be trained, for example,based on the methods described in FIG. 3 or 4, according to therequirements of different users. Each of the plurality of machinelearning models may be configured to determine that the specifiedrequirements used to evaluate an electronic document applied to one ormore identified features of the electronic document indicates anunintentional bias. The indications of unintentional bias may beidentified and mitigated in subsequent electronic document evaluation.For example, items or patterns previously identified by experts as beingtriggers for bias may be correlated with a value associated with anidentified feature and a score of potential bias may be suggested,thereby mitigating the effects of subtle biases on the evaluation.

At block 620, an electronic document may be evaluated by the pluralityof neural networks. Each of the plurality of neural networks (e.g., theneural networks 510 a-510 n) may execute a machine learning model. Insome embodiments, at least one of the plurality of machine learningmodel may be trained using training data based on common requirements,for example, but not limited to, minimum education level, minimum skilllevel for a category of positions, etc., applicable to the other machinelearning models of the plurality of machine learning models. Theelectronic document may be evaluated sequentially, first by the machinelearning model trained using training data based on common requirements,then substantially concurrently by the other machine learning models.

At block 630, an activity may be performed to obtain additionalinformation from a candidate. Each of the neural networks may execute amachine learning model to evaluate electronic document. Based on theevaluations, one or more of the neural networks may output aspects ofthe electronic documents requiring additional information may beidentified to the computer system (e.g., the computer system 520). Forexample, a machine learning model may evaluate a resume and identifygaps in employment history or other missing information. In some cases,a candidate may be asked to supply information regarding something notfound on the resume but is found on other similar resumes. The computersystem may perform an activity to obtain the additional information. Forexample, the computer system may generate an information request (e.g.,an email) to the candidate that submitted the resume to obtain therequired additional information.

At block 640, the electronic document may be reevaluated including theadditional information. When the requested additional information isreceived by the computer system (e.g., by a return email from thecandidate), the computer system may output the additional information asadditional training data for the at least one of the plurality ofmachine learning models that identified the aspect requiring the furtherinformation and the electronic document may be reevaluated using themachine learning model modified with the additional training dataprovided by the additional information.

At block 650, the neural networks may output a value or indicationregarding the electronic document. Each of the machine learning modelsmay generate a value or an indication for the electronic document andthe neural networks may output the value or indications. The value orindication may indicate a degree of compliance with the requirements ofa specific user.

In some embodiments, at block 660, the outputs of each of the pluralityof neural networks may be combined. In accordance with various aspectsof the present disclosure, the outputs of the plurality of neuralnetworks may be combined in a “committee of machines” approach toevaluating electronic documents. Referring to FIG. 5, a combiner 530 maybe included in the system 500 to combine the outputs 515 a-515 n of theneural networks 510 a-510 n. The combiner 530 may average or weight theoutputs of the neural networks by applying averaging or weightingalgorithms known to those of skill in the art. The combiner 530 mayoutput 532 the combined outputs of the neural networks to the computer520. Referring again to FIG. 6, at block 670, a value or indication maybe generated based on the combined outputs of the neural network modelsand output 524 by the computer 520. The value or indication may indicatea degree of compliance with the combined requirements of the users.

In accordance with various aspects of the present disclosure, when a newresume is received the machine learning models may evaluate the contentand through the committee of machines approach provide a value orindication for the resume for each machine learning model and also forthe combined machine learning models. The value or indication mayindicate a degree of compliance with requirements of the individualusers and the combined requirements of the users.

It should be appreciated that the specific steps illustrated in FIG. 6provide a particular method for evaluating electronic documentsaccording to another embodiment of the present invention. Othersequences of steps may also be performed according to alternativeembodiments. For example, alternative embodiments of the presentinvention may perform the steps outlined above in a different order.Moreover, the individual steps illustrated in FIG. 6 may includemultiple sub-steps that may be performed in various sequences asappropriate to the individual step. Furthermore, additional steps may beadded or removed depending on the particular applications. One ofordinary skill in the art would recognize many variations,modifications, and alternatives.

The examples and embodiments described herein are for illustrativepurposes only. Various modifications or changes in light thereof will beapparent to persons skilled in the art. These are to be included withinthe spirit and purview of this application, and the scope of theappended claims, which follow.

What is claimed is:
 1. A method for providing training data for amachine learning model, the method comprising: obtaining a selectionfrom equipment of a specific user responsive presentation of proposedwords according to an A/B test; identifying, by application of theselection to a dedicated neural network, a pre-trained, baseline machinelearning model; displaying a plurality of electronic documents forevaluation on a display of a computing device to obtain a displayedplurality of electronic documents; tracking, by an eye movement trackingdevice, a gaze of the specific user as the specific user reads each ofthe displayed plurality of electronic documents on the display todetermine indications of pauses in reading each of the displayedplurality of electronic documents for greater than a specified period oftime by the specific user, the pauses indicative of the specific user'sgaze; correlating objects on each of the displayed plurality ofelectronic documents to the pauses in reading by the specific user;identifying features for the machine learning model to obtain identifiedfeatures based on the objects and textual analysis of each of theplurality of electronic documents; presenting information related toeach identified feature to the specific user; obtaining from thespecific user a descriptor defining each of the identified features anda value for each of the identified features indicating a relativeimportance or applicability of each of the identified features;associating obtained descriptors and values with each of the identifiedfeatures; obtaining from the specific user an overall value for each ofthe plurality of electronic documents indicating an overallapplicability of each of the plurality of electronic documents withrespect to specific requirements of the specific user; associatingobtained overall values with each of the plurality of electronicdocuments; and combining the pre-trained, baseline machine learningmodel, identified features, associated descriptors, associated values,and associated overall values as training data for the machine learningmodel associated with the specific user.
 2. The method of claim 1,wherein a plurality of same features and a plurality of differentfeatures are identified among the plurality of electronic documents. 3.The method of claim 1, further comprising: determining that a valueassociated with an identified feature is an indication of bias; andidentifying the indication of bias in the training data for mitigationduring subsequent electronic document evaluation.
 4. The method of claim1, wherein the plurality of electronic documents comprise jobapplications or resumes; and the features comprise one or more oflogical skill development information, career progression information,misleading phrases related to employment experience, and gaps inemployment history.
 5. The method of claim 1, wherein the objects areone of a word, a phrase, and a picture.
 6. The method of claim 1,wherein the determining indications of pauses in reading comprises:tracking, by the eye movement tracking device, eye positions of thespecific user's eyes on each of the displayed plurality of electronicdocuments as the specific user reads each of the displayed plurality ofelectronic documents; and detecting pauses in movement of the specificuser's eyes for greater than the specified period of time on a displayedelectronic document or detecting repeated eye movements betweendifferent portions of the displayed electronic document.
 7. The methodof claim 6, wherein the correlating of the objects on each of thedisplayed plurality of electronic documents to the pauses in readingcomprises: determining, based on data from the eye movement trackingdevice, the eye positions of the specific user's eyes on the display;correlating portions of each of the displayed plurality of electronicdocuments with the eye positions of the specific user's eyes; andidentifying the objects corresponding to the portions of each of thedisplayed plurality of electronic documents.
 8. The method of claim 1,wherein the display is a touch-sensitive display, wherein the eyemovement tracking device comprises the touch-sensitive display, whereinthe specific user follows text of each of the displayed plurality ofelectronic documents across the touch-sensitive display with a finger ora stylus in contact with the touch-sensitive display as the specificuser reads each of the displayed plurality of electronic documents, andwherein a determining of the indications of pauses in reading comprisesdetermining, based on data from the touch-sensitive display, pauses inmovement of the finger or the stylus in contact with the touch-sensitivedisplay for the specified period of time.
 9. The method of claim 8,wherein the correlating an object on each of the displayed plurality ofelectronic documents to the pauses in the specific user's readingcomprises: determining, based on the data from the touch-sensitivedisplay, positions on the touch-sensitive display of the finger or thestylus in contact with the touch-sensitive display; correlating portionsof each of the displayed plurality of electronic documents with thepositions of the finger or the stylus; and identifying the objectscorresponding to the portions of each of the displayed plurality ofelectronic documents.
 10. The method of claim 1, wherein the display isa touch-sensitive display configured to provide haptic feedback inresponse to a specified amount of pressure exerted on the display by afinger or a stylus, wherein the specific user follows text of each ofthe displayed plurality of electronic documents with the finger or thestylus as the specific user reads each of the displayed plurality ofelectronic documents, and wherein a determining of the indications ofpause in reading comprises exerting the specified amount of pressure onthe touch-sensitive display with the finger or the stylus to generatethe haptic feedback for the specified period of time.
 11. The method ofclaim 10, wherein the correlating of the objects on each of thedisplayed plurality of electronic documents to the pauses in readingcomprises: determining, based on data from the touch-sensitive display,positions of the finger or the stylus exerting the specified amount ofpressure on the touch-sensitive display to generate the haptic feedbackfor the specified period of time; correlating portions of each of thedisplayed plurality of electronic documents with the positions of thefinger or the stylus; and identifying the objects corresponding to theportions of each of the displayed plurality of electronic documents. 12.The method of claim 1, further comprising identifying training data fora plurality of different models, each of the plurality of differentmodels trained based on training data identified for a different user.13. A system, comprising: a plurality of machine learning models, eachof the plurality of machine learning models trained with initialtraining data generated by a training method including, for each machinelearning model: obtaining a selection from equipment of a specific userresponsive presentation of proposed words according to an A/B test;identifying, by application of the selection to a dedicated neuralnetwork, a pre-trained, baseline machine learning model; displaying aplurality of electronic documents for evaluation on a display of acomputing device to obtain a displayed plurality of electronicdocuments; tracking, by an eye movement tracking device, a gaze of thespecific user as the specific user reads each of the displayed pluralityof electronic documents on the display to determine indications ofpauses indicative of the gaze of the specific user in reading each ofthe displayed plurality of electronic documents for greater than aspecified period of time by the specific user; correlating objects oneach of the displayed plurality of electronic documents to the pauses inreading by the specific user; identifying features for the machinelearning model to obtain identified features based on the objects andtextual analysis of each of the plurality of electronic documents;presenting information related to each identified feature to thespecific user; obtaining from the specific user a descriptor definingeach of the identified features and a value for each of the identifiedfeatures indicating a relative importance or applicability of each ofthe identified features; associating obtained descriptors and valueswith each of the identified features; obtaining from the specific useran overall value for each of the plurality of electronic documentsindicating an overall applicability of each of the plurality ofelectronic documents with respect to specific requirements of thespecific user; associating obtained overall values with each of theplurality of electronic documents; and combining the pre-trained,baseline machine learning model, identified features, associateddescriptors, associated values, and associated overall values astraining data for the machine learning model associated with thespecific user; and a processing system including a processor configuredto accept outputs from each of a plurality of neural networks executingthe plurality of machine learning models and perform specifiedactivities based on the outputs of each of the plurality of neuralnetworks, wherein each machine learning model in a subset of theplurality of machine learning models is trained using training databased on requirements of a different user, wherein at least one of thesubset of the plurality of machine learning models evaluates aparticular electronic document, and based on an evaluation identifies atleast one aspect of the particular electronic document requiringadditional information, and wherein an output of the at least one of theplurality of neural networks causes the processing system to perform anactivity to obtain the additional information related to the identifiedat least one aspect of the particular electronic document.
 14. Thesystem of claim 13, wherein the particular electronic document comprisesa job application or a resume and the additional information relates toone or more of inconsistent logical skill development information,career progression information, and gaps in employment history.
 15. Thesystem of claim 14, wherein the activity performed by the processingsystem caused by an output of the at least one of the plurality ofneural networks comprises contacting a candidate that submitted the jobapplication or resume with a request to provide the additionalinformation related to the identified at least one aspect.
 16. Thesystem of claim 15, wherein when the additional information is receivedby the processing system, the processing system outputs the additionalinformation as additional training data for the at least one of theplurality of machine learning models, and wherein based on the initialtraining data and the additional training data, the at least one of theplurality of machine learning models is configured to generate a valueor an indication for the particular electronic document.
 17. The systemof claim 13, wherein the electronic documents comprise one or more ofjob applications, resumes, and published articles.
 18. The system ofclaim 17, wherein the activity performed by the processing system causedby an output of the at least one of the plurality of neural networkscomprises contacting authors of the published articles or candidatesthat submitted the job applications or the resumes.
 19. The system ofclaim 13, wherein at least one of the plurality of machine learningmodels not included in the subset of the plurality of machine learningmodels is trained using training data based on requirements applicableto the subset of the plurality of machine learning models, and whereinthe electronic documents are evaluated sequentially by the at least oneof the plurality of machine learning models and then by the at least oneof the subset of the plurality of machine learning models.
 20. Thesystem of claim 13, wherein each of the plurality of machine learningmodels is configured to determine that the specific requirements of thespecific user used to evaluate an electronic document applied to one ormore identified features of the electronic document indicates a bias;and mitigate the bias during subsequent evaluation of the electronicdocument.
 21. A method for evaluating electronic documents using aplurality of machine learning models, the method comprising: trainingthe plurality of machine learning models for a plurality of differentspecific users by a training method including, for each machine learningmodel: obtaining a selection from equipment of a specific userresponsive presentation of proposed words according to an A/B test;identifying, by application of the selection to a neural network, apre-trained, baseline machine learning model; displaying a plurality ofelectronic documents for evaluation on a display of a computing deviceto obtain displayed plurality of electronic documents; tracking, by aneye movement tracking device, a gaze of the specific user as thespecific user reads each of the displayed plurality of electronicdocuments on the display to determine indications of pauses in readingeach of the displayed plurality of electronic documents for greater thana specified period of time by the specific user, wherein the pauses areindicative of the gaze of the specific user; correlating objects on eachof the displayed plurality of electronic documents to the pauses inreading by the specific user; identifying features for the machinelearning model to obtain identified features based on the objects andtextual analysis of each of the plurality of electronic documents;presenting information related to each identified feature to thespecific user; obtaining from the specific user a descriptor definingeach of the identified features and a value for each of the identifiedfeatures indicating a relative importance or applicability of each ofthe identified features; associating obtained descriptors and valueswith each of the identified features; obtaining from the specific useran overall value for each of the plurality of electronic documentsindicating an overall applicability of each of the plurality ofelectronic documents with respect to specific requirements of thespecific user; associating obtained overall values with each of theplurality of electronic documents; and combining the pre-trained,baseline machine learning model, identified features, associateddescriptors, associated values, and associated overall values astraining data for the machine learning model associated with thespecific user; evaluating the electronic documents by each machinelearning model of the plurality of machine learning models based onrequirements of the specific user of the plurality of different specificusers used to train each of the plurality of machine learning models;outputting a value or an indication for the electronic documents by eachneural network executing one of the plurality of machine learningmodels; combining outputs of each neural network with a combiner; andgenerating a final a score or indication indicating a degree ofcompliance with combined requirements of the plurality of differentspecific users based on the combined outputs of the plurality of neuralnetworks.
 22. The method of claim 21, wherein each of the plurality ofmachine learning models is configured to determine that a valueassociated with an identified feature is an indication of bias; andmitigate the bias during subsequent evaluation of the electronicdocuments.
 23. The method of claim 21, wherein at least one of theplurality of machine learning models is trained based on requirementsapplicable to each other machine learning model of the plurality ofmachine learning models, and wherein the electronic documents areevaluated sequentially by the at least one of the plurality of machinelearning models and then by the each other machine learning model of theplurality of machine learning models.
 24. The method of claim 23,wherein the electronic documents are resumes and the one or morerequirements are employment qualifications.
 25. The method of claim 23,wherein the requirements applicable to the each other machine learningmodel of the plurality of machine learning models are general employmentqualification for an organization.
 26. A method for providing trainingdata for a machine learning model, the method comprising: obtaining aselection from equipment of a specific user responsive presentation ofproposed words according to an A/B test; identifying, by application ofthe selection to a dedicated neural network, a pre-trained, baselinemachine learning model; displaying a plurality of electronic documentsfor evaluation on a display of a computing device to obtain a displayedplurality of electronic documents; tracking, by an eye movement trackingdevice, a gaze of the specific user as the specific user reads each ofthe displayed plurality of electronic documents on the display todetermine indications of pauses in reading each of the displayedplurality of electronic documents, wherein the pauses are indicative ofthe gaze of the specific user; annotating electronic documents displayedon a display of a computing device with one or more annotations;correlating objects on the electronic documents to the gaze of thespecific user and at least one of the one or more annotations;identifying features for the machine learning model to obtain identifiedfeatures based on the gaze of the specific user and annotated objectsand textual analysis of each of the electronic documents; presentinginformation related to each identified feature to the specific user;obtaining from the specific user a descriptor defining each of theidentified features and a value for each of the identified featuresindicating a relative importance or applicability of each of theidentified features; associating obtained descriptors and values witheach of the identified features; obtaining from the specific user anoverall value for each of the plurality of electronic documentsindicating an overall applicability of each of the plurality ofelectronic documents with respect to specific requirements of thespecific user; associating obtained overall values with each of theplurality of electronic documents; and combining the pre-trained,baseline machine learning model, identified features, associateddescriptors, associated values, and associated overall values astraining data for the machine learning model associated with thespecific user.
 27. The method of claim 26, wherein the display is atouch-sensitive display, and wherein the annotating the electronicdocuments comprises marking up the electronic documents on thetouch-sensitive display with a finger or a stylus using consistentmark-up symbols having defined meanings.
 28. The method of claim 26,wherein the annotating the electronic documents comprises making audioannotations using a speech recognition engine.